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ABSTRACT 



This paper reviews the 1 i terature pertinent to 



learning with graphics. The dual coding theory provides explanation 
about how graphics are stored and precessed in semantic memory* The 
level of processing theory suggests how graphics can be employed in 
learning to encourage deeper processing. In addition to dual coding 
theory and level of processing theory, the area of motivation and 
attention is aiso reviewed to show how graphics and emotion are 
related and how graphics should be designed to communicate desired 
intent. It is noted that appropriate design decisions about the use 
of graphics are often based on knowledge from cognitive science and 
that current research in graphics design related to this topic is 
inadequate. The area of cognitive science is incorporated into 
instructional technology so that the effectiveness of graphics in 
assisting can be optimized. (Contains 48 references.) (Author/MES) 
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Abstract : 

The purpose of this paper is to review literature pertinent to learning with 
graphics. The dual coding theory provides explanation about how graphics are 
stored and processed in semantic memory. The level of processing theory suggests 
how graphics can be employed in learning to encourage deeper processing. In 
addition to dual coding theory and level of processing theory, the area of 
motivation and attention will also be reviewed in this paper to show how graphics 
and emotion are related and how graphics should be designed to communicate 
desired intent. As noted, appropriate design decisions about the use of grapliics 
are often based on knowledge from cognitive science. However, current research 
in graphics design related to this topic is inadequate. This article attempts to 
incorporate the area of cognitive science into instructional technology so that the 
effectiveness of grapliics in assisting can be optimized. 
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A Theoretical analysis of Visual Learning 
- Implications for Computer Graphics Design 

Lih-Juan ChanLin 
Department of Library & Information Science 
Fu-Jen University 

Gagne, Briggs, & Wager (1992) suggested that the instruction should be 
designed not only to incorporate the principles of behaviorism but should also 
consider how information is processed in memory. To employ graphics in the 
design of computer assisted instruction, it is necessary to understand how graphics 
are cognitively processed. Although the effects of graphics have not been 
extensively studied and researched in computer assisted instruction, graphics have 
been frequently used in the design of various instructional materials and media for 
communication. 

The use of graphics in communication has had a long history. Through art and 
technology, human beings have been engaged in reconstructing their perceptual 
world for thousands of years (Bolter, 1991). Ever since human beings learned to 
communicate, symbols have been created and used to communicate. Pictographic 
language evolved in some cultures, such as Chinese and Egyptian. For example, the 
Chinese character was a graphic symbol depicting an object or event by the people 
within a homogeneous culture group, certain pictorial or verbal information gives 
rise to similar interpretations of a specific reality or message (Pettersson, 1989). 

Like communication through verbal language, the use of graphics relies on the 
recall of a visual memory (Mealing & Yazdani, 1990). Pictures can communicate 
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more than just their surface contents because a wide range of prior knowledge and 
experience is brought to bear on the imagery by the viewers, and at the same time, 
this can be exploited to enrich the meaning of an graphic. Although computer 
technology does not have a long history, computer graphics, animation, and various 
visual effects have been widely incorporated in instructional design. Much 
instructional design is based on designer's intuition about what should be done. In 
this paper, some of the theories that related with the use of graphics will be 
discussed. The dual coding theory discusses the natural of semantic memory. The 
level of processing focuses on the processes learners employed in encoding with 
graphics. Motivation and attention suggest the emotional factors related with the 
process of deciphering and encoding of given information. In addition, the 
implications for graphics design are discussed. 

Coding Systems 

The best support for the merits of graphics comes from dual coding theory 
(Paivio, 1971; 1983; 1986; 1990; 1991). The primary focus of dual coding theory 
concerns the nature of semantic representation in memory. According to Paivio, 
there are two functionally distinguishable symbolic systems in semantic memory - 
one verbal and the other non-verbal. Dual-coding theory suggests that when 
information is dual-coded, it is easier to retain in memory. The important points 
emphasized in this theory are that the verbal and non-verbal systems can be 
activated independently and that there are interconnections between the systems that 
allows dual codes. It is assumed that concrete information can be stored in either 
non-verbal image system, verbal system, or both systems. 

The two systems have different organizational and processing characteristics. 
The basic units in the verbal system are termed logogensr, and the basic units in the 
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non-verbal systems are termed imagens. Sadoski, Paivio, & Goetz (1991) 
compared the structures of the two systems. They described that information in the 
verbal system is organized with a sequential or hierarchical structure; while non- 
verbal information is organized within a nested set of structure (see Figure 1). In 
Figure 1, the structures of verbal system and non-verbal system are contrasted in 
the example. In the verbal system, an eyeball would be described by arranging 
information into words and sentences, an eye would be described through the 
formation of paragraphs, and the descriptions of an entire face might compose a 
chapter. Unlike the verbal system, the non-verbal system is not structured 
sequentially . For example, eyes, nose , and mouth are perceived separately, but 
are parts of a face. Even the eye is a nested set, having an eyelid, and an eyeball. 
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Dual coding theory distinguishes three different levels of processing: 
representational processing, referential processing, and associative processing. 
Representational processing refers to the activation of logogens by verbal stimuli 
and imagens by nonverbal stimuli. For example, the word "dog" activate the 
corresponding logogen, and the picture of a dog activate corresponding imagen. 
Associative processing refers to intra-system (within image or verbal system) 
connections. In other words, words or phrases are associated with other words or 
phrases. For example, the phrase "once upon a time" may activate the word 
"prince" or "princess". The phrase "once upon a time" may activate other ■ 
language units based on people's experience. Referential processing refers to the 
inter-system (cross system) activation. For example, language can evoke image, 
and imagery can evoke language as well. 

According to Paivio, all cognitive tasks require representational processing and 
some may involve all three levels of processing. In other words, when viewing the 
word "apple", under the representational processing, the corresponding descriptions 
(logogens) which directly associated with fruit apple will be activated. Under 
associative processing, the fruit orange might be activated because apple and orange 
are under the same class fruit. However, if a person is working extensively with an 
Apple computer, at associative processing level, the description associated with an 
Apple computer might be activated under appropriate contextual conditions. At 
referential processing level, an image of apple or an image of an Apple computer 
will be activated by the descriptions of an apple or an Apple computer. A notation 
might be used as followed to express the above dcscnptions about the three levels of 
processing: apple -> representational processing s descriptions of apple -> 
associative processing -> descriptions of an orange or an Apple computer (based on 
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the context) -> referential processing -> an image of an orange or an image of an 
Apple computer. 

Either verbal or visual information can trigger activity in any of three levels of 
processing within or cross systems. The verbal system and non-verbal system are 
bi-directionally interconnected, and the interconnection is assumed to be one-to- 
many, rather than one-to-one. For example, a given picture can evoke any of a 
number of verbal or image information, or descriptions corresponding to different 
exemplars of a referent class. Precisely which images or descriptions will be 
activated at any moment depends on the context of the stimuli, and how those 
stimuli interact with viewer's experiences and expectations. 

Paivio postulated that pictures were easier to remember than words due to "the 
mnemonic superiority of the image code over the verbal code" (Paivio, 1991, pp 
265). Although it is assumed that pictures and words activate independent imaginal 
and verbal codes, the availability of this dual code differs. That is, a verbal code 
for a picture is more available than an imaginal code for a word. As stated, 
pictures were better remembered than words because pictures were more likely to 
be redundantly (additively) encoded. There is evidence to support this assumption. 
For example, Paivio and Csapo (1973) investigated the relative merits of imaginal 
and verbal codes using an incidental recall task in wliich the orienting task was 
given to control the way items are coded during input. The results indicated that 
higher recall test scores were obtained in imaginal codes. In another study, Paivio 
(1975) also concluded from a repetition-lag experiments that verbal and non-verbal 
memory codes are statistically independent and additive in their effect on free 
recall, however, the non-verbal is mnemonically stronger than the verbal codes. 
Other studies such as Paivio & Lambert (1981), Roediger & Weldon (1987), and 
Weldon & Roediger (1987) also had the similar findings. 
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In addition to addressing issues of encoding, Paivio (1991) provided guidelines 
for maximizing retrieval of specific content that has been learned. As suggested, 
effective retrieval cues activate the representations created at encoding. If the 
information is coded with verbal or pictorial cues, more correct recall would occur 
when encoding cues are used in retrieval task. Picture superiority effect can be 
reversed by changing retrieval demands (see also Roediger & Weldon, 1987; 
Weldon & Roediger, 1987). This proposition is similar to Tulving's encoding 
specificity theory of retrieval (1983). 

Although there are criticisms of the dual coding theory (Paivio, 1991; 
Kobayshi, 1986), the dual coding theory is to date the dominant theory tot directly 
addresses the role of images in cognition. Because the semantic interconnections 
between verbal system and image system are assumed to be one-to-many, rather 
than one-to-one, in both directions. It is worth noticing that a given picture can 
evoke any of a number of verbal (or image) information or descriptions 
corresponding to different exemplars of a referent class. In order to design 
effective graphic materials, the characteristics and experiences of learners in 
processing the cues should be well understood, and the use of pictorial cues in 
retrieving is also need to be consistent with the cues used at encoding. 

Coding Processes 

Another theory that supports the need for the use of pictures in learning and 
memory is "Level of Processing" theory (Craik & Lockhiit, 1972). As noted by 
Paivio (1991), the superior recall effects of the dual coding of to-be-remembered 
information might be explained by levels of processing (or elaborative processing). 
According to level of processing theory, the deeper the information is processed, 
the longer it will be retained in memory. In other words, if the information to be 
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remembered is rehearsed with specific mnemonic strategy or cueing materials, it 
can retain longer in memory. 

To provide opportunities for deep processing (or elaborative processing), the 
use of imagery instruction can be used to encourage the deep processing by learners 
of to-be-learned materials. For example, in the area of reading instruction for 
children, imagery instruction has been widely used as a rehearsal strategy for 
remembering what was read. In Figure 2, the Mother Goose's rhyme on the left 
can be presented to children. The children can be asked to imagine a picture about 
the story, or they can draw a picture about the story in order to help them 
remember the rhyme and the story. Although imagery instructions facilitate 
memory of to-be-learned materials, a lot of training and practicing is often 
required (Pressley, 1977; Alesandrini, 1982; Pressley, Cariglia-Bull, Deane & 
Schneider, 1987). The difficulties in carrying out imagery instruction might 
overtax children's functional memory limitations, and it was concluded that using 
imagery often does not facilitate young children's verbal learning (Pressley & 
Levin, 1977, 1978). As noted by Pressley, Cariglia-Bull & Deane (1987), the 
effectiveness of imagery instruction is related with individuals' intellectual 
competence, their memory capacity and experience with the strategy. In regard to 
learning of new concept, the imagery instruction might not be successful because 
we cannot expect people to form mental images of concepts not yet known, 
especially when a complex science concept or math concept are involved. In order 
to use an imagery strategy successfully, appropriate training and instruction are 
required. The imagery instruction might be varied for different group of learners 
or different individuals due to differences in their memory capability. 

In contrast to generating images internally by learners for rehearsing the 
learning content as mentioned above, providing pictures externally from 
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There was an old woman tossed in a 
basket 

Seventeen tunes as high as tfie moon; 
For under her arm she carried a 
broom 

Old woman, old woman, said I! 
Whither, oh whither, oh whither so 
high? 

To sweep the cobwebs from the sky, 
And Vll be with you by and by. 



* 




Figure 2. Mother Goose's Nursery Rhyme - There was an old woman tossed 
in a basket 



instruction contributes substantial merit to help learners visualize the concepts to be 
learned (Mayer & Anderson, 1992; Rieber, 1994). Pictures have been commonly 
used to help learning within different content areas. When presented with the 
reading material, pictures facilitated children's retention and recall of Hie reading 
materials (Goldstein & Underwood, 1981; Haring & Fry, 1979; Levie & Lentz, 
1982). Graphics provide more cues to allow deeper processing of information in 
their memory. For example, in Figure 2, the grapliics on the right allows children 
to compare every detail in the picture with the words or sentences in the rhyme so 
that the story and the rhyme can be remembered through rehearsing with these 
cues. 

In studying the effect of grapliics in learning science instruction, Reid, Briggs & 
Beveridge (1983) found significant improvement in post-test scores among subjects 
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when pictures are presented. As observed, due to the increased attention among 
individuals toward the materials, learners are motivated to compare more carefully 
the information available in pictures and written prose. This rehearsing or 
elaborative process leads to the development of more effective semantic storage 
networks so that the information can be easily retrieved later. It might be possible 
to describe this effect in terms of Craik & Lockhart's level of processing (1972) of 
the given information and hence facilitate recall. 

Although level of processing theory argues deep process and elaborative process 
encourages better recall, it emphasizes on semantic level, rather than sensor level 
of processing. According to level of processing view, memory traces are "by- 
products" of the perceptual processing (Kobayshi, 1986). The perceptual 
processing involves analysis from incoming stimuli from meaningful information. 
In other words, viewers tend to associate given pictures with text based on their 
own interpretation of which provides meaningfulness to them. Under this 
argument, it is worthwhile to notice whether the pictures provide necessary 
connections for learners to associate between what they are familiar with and what 
they need to learn or remember. In figure 2, when reading the rhymes with the 
pictures, learners might tend to relate each sentence either with the whole picture 
or with specific portion of the picture. They might look for the cues in the pictures 
to help them construct the meaning described in the rhyme so that the materials can 
be easily processed and coded in their memory. For example, when they read the 
word "cobweb", they might look for the clue in the picture. When they are not 
able to find a real "cobwebs", they might interpret the cobweb-like star rays to 
what has been referred in the story. 

The level of processing theory provides an explanation of the process in 
learning with graphical information. Individual differences, which presumably 
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reflect innate factors and the experiences of the individual, influence the activation 
of representations and the effect of specific stimulus attributes. These are all 
important considerations for graphic design. Understanding the characteristics of 
the learners and how learners associate between the text and pictures is essential. 

Motivation and Attention 

In the real world, emotion is closely related to the process of learning (Zajonc, 
1980). Emotion affects how information is attended, stored and retrieved 
(Langhinrichsen & Tucker, 1990; Lehmann & Koukkou,1990). Since perceiving 
information involves cognitively selecting and filtering the elements from given 
stimuli (Pettersson, 1989), the learners' interests and expectations play an important 
role in extracting information and reconstructing their understanding. 

The emotional states of learners is considered an important variable in 
determining the effectiveness of learning information. Motivation refers to arousal 
and goal-oriental aspects of behavior that company the verbal or nonverbal 
memory system, and affective reactions would ordinarily occur more quickly to 
pictures than to words (Paivio, 1986; 1990). 

Rogers (1983) considers emotion as a third code of cognition in addition to 
verbal codes and imagery codes. He reveals that how people feel is related with how 
people can learn and remember from given information. What makes a piece of 
information fit into cognition is largely a matter of coherence of one's belief and 
desire. Implied from Rogers' interpretation, the emotional effects that graphics 
can provide is highly related to how given pictures are processed and how they 
interact with learners' internal mental states. As proposed, affective reaction to the 
graphic information is part of the encoding process in cognitive system (Roger, 
1983). He assumed that when affective response is activated, retention is enhanced. 
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The positive attitude toward the learning materials motivates learners actively 
operate and organize the given representations into their own construction. 

When related with what learners like and what make them curious to see, 
pictures might create considerable learning interest that motivate them to learn the 
information described in the text (Samuels, 1970). When individuls are interested, 
it is more likely that they will devote their effort in learning tasks (Keller, 1983; 
Visser & Keller, 1990). Although how computer graphics motivate students to 
learn has not been actually explored and measured from empirical studies, a lot of 
empirical studies (Levie & Lentz, 1982; Mayer, 1989; Mayer & Anderson, 1992; 
Mayer & Gallini, 1990; Rieber, 1990; Swell & Moore, 1980) have addressed the 
emotional impact on learning. The finding suggests that pictures enhance readers' 
enjoyment, and help children develop positive attitudes toward reading and make 
the task of learning to read more pleasant. When involvement is enhanced, it 
becomes easier for students to benefit from the reading materials and acquire 
concepts they need to know from the verbal materials. 

Given that affective reactions would ordinarily occur more quickly to pictures 
than to words (Paivio, 1990), more precautions and considerations should be taken 
in relating what is perceived by learners and what is desired to be perceived from 
the intended objective. Fitzgerald (1989) argued that the use of pictures needs to 
more attend to the properties in the pictures relevant to establish intended belief and 
desires from learners. Due to different people have different experiences with 
graphics, when a picture is used to communicate, considerations should be taken 
regarding whether the picture is appropriate to learners' experiences. For example 
, the use of red color might create warm feeling. It might also be interpreted as a 
"warning" or "wrong response" by viewers. The design decision about integrating 
specific graphic attributes, such as color, format, or animation, depends not only on 
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"which" attribute to choose to interact with learners but also "how" to use it 
properly and "when" to use it to meet intended purposes. 

Implications from Graphic Research 

Psychologically, pictures provide distinct features in presenting intended 
information and facilitate certain emotional reactions. As noted by Salomon 
(1979), pictures can communicate better only when the symbolic codes they use are 
closer to, or more congruent with what receivers have experienced or what ought 
to generate for concept construction. In order to use graphic materials to help 
students learn, one should be able to incorporate graphics related to the concepts or 
knowledge they have learned and they are going to leam . 

The instructional justifications of using visualization in various intended 
purposes including increasing learning interest, motivation, curiosity; presenting 
opportunities for learner to perceive intended context; spanning linguistic barriers; 
fostering generalization of responses to new situations; providing greater flexibility 
in the organization of instruction; illustrating, clarifying, and reinforcing oral and 
printed communication, and sharpening powers of observation (Dwyer, 1985). 
However, graphics might not successfully facilitate learning performance in 
different learning setting. Although the use of graphics is integrated in designing 
instruction, many learners are still unable to acquire the intended information 
through given graphics if they fail to actively incorporate elaboration strategies for 
learning (Rieber, 1994). It is worthwhile to examine whether the used graphics 
accurately illustrate and clarify the content material being presented. It is also 
essential to carefully study how the information can be constructed and perceived 
by individuals based on their cognitive ability, characteristic, learning experience 
and knowledge background. 
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Literature has addressed that individuals benefit from graphics differently due 
to several factors. For example, task difficulty for specific group of learners 
(Dwyer & Parkhurst, 1984), levels of 1Q among learners (Berry & Dwyer, 1982; 
Dwyer, 1983), reading ability of learners (Reid & Beveridge, 1986; Samuels, 
1970), different graphic complexity (Dwyer & Parkhurst , 1984), and presenting 
strategies (e.g. animation) used with graphics (Mayer & Anderson, 1992; Rieber, 
1989) are considerations for using graphics. 

During recent years, there has been a growing interest and development effort 
for graphical innovations, such as three-D and virtual reality technology. These 
technical advances make the communication of the information available in a wide 
variety of approaches. To design effective visual stimuli used in computer assisted 
instruction, it is necessary to investigate specific cognitive level, characteristic, and 
match stimuli design with specific kind of learners. For example, should visual 
displays be simple or more complex, what colors and cues should be used, and in 
what sequence should these graphics be presented. Questions of this type should 
then be compared with specific cognitive ability and to a specific learning outcome 
so that the learners with specific cognitive ability which may have information 
processing disadvantages might be improved through the use of graphics. 

Summary 

The above review addresses implications for the use of graphics in computer 
assisted instruction such as the treatment of graphics in relation to the type of 
learning tasks given, metacognitive skills, the learning patterns of learners, and the 
level of students' ability to functionally use the given representation to incorporate 
into their own construction of knowledge. These issues addressed emphasize the 
importance of carefully study about how learner learn from graphics prior to 
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design work. Dual coding theory and level of processing theory regarding learning 
from graphics suggest the differences in learner's processing ability and process 
patterns can be factors accounting for the effect of using computer graphics. 
Individual differences are also important considerations in employing graphics for 
motivation and attention purposes. Whether the computer graphics helps or retards 
the thinking process of in the learning context depends on whether the learner can 
use the graphically representing ideas to integrate the main concept of the content 
and, thus, to form the cognitive connections. With this implication, the use of 
computer graphics should take into considerations of the relation provided in 
linking the cognitive components: attending, coding, and processing that function as 
an anchoring schema in helping learners reconstructing and extracting the learning 
concepts rather than just an aid for decorating computer screens. 

In order to provide reliable support for using computer graphics, considerably 
more research and development in exploring graphics in different cognitive 
acquisition are desired. The process of design computer graphics should start with 
the process of understanding how and why learners learn from graphics and then 
trying to project the key to mastering the learning tasks from the graphics designed. 
Although the advance technology has provided graphic designers with various new 
design opportunities, the central element to successful graphic design lies on the 
understanding of how learners learn from graphics. 
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